System and method for dynamic transparent consistent application-replication of multi-process multi-threaded applications

ABSTRACT

A system, method, and computer readable medium for consistent and transparent replication of multi process multi threaded applications. The computer readable medium includes computer-executable instructions for execution by a processing system. Primary applications runs on primary hosts and one or more replicated instances of each primary application run on one or more backup hosts. Replica consistency between primary application and its replicas is provided by imposing the execution ordering of the primary on all its replicas. The execution ordering on a primary is captured by intercepting calls to the operating system and libraries, sending replication messages to its replicas, and using interception on the replicas to enforce said captured primary execution order. Replication consistency is provided without requiring modifications to the application, operating system or libraries.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/826,659 filed Aug. 14, 2015 titled SYSTEM AND METHOD FOR DYNAMICTRANSPARENT CONSISTENT APPLICATION-REPLICATION OF MULTI-PROCESSMULTI-THREADED APPLICATIONS, issued as U.S. Pat. No. 9,405,633 on Aug.2, 2016, which is a continuation of U.S. application Ser. No. 12/887,208filed Sep. 21, 2010 titled SYSTEM AND METHOD FOR DYNAMIC TRANSPARENTCONSISTENT APPLICATION-REPLICATION OF MULTI-PROCESS MULTI-THREADAPPLICATIONS, issued as U.S. Pat. No. 9,135,127 on Sep. 15, 2015, whichis a continuation-in part and claims priority from U.S. application Ser.No. 12/851,706 filed Aug. 6, 2010 titled SYSTEM AND METHOD FORTRANSPARENT CONSISTENT APPLICATION-REPLICATION OF MULTI-PROCESSMULTI-THREADED APPLICATIONS, issued as U.S. Pat. No. 8,589,953 on Nov.19, 2013, the disclosures of which are incorporated herein by referencein their entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable

NOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION

A portion of the material in this patent document is subject tocopyright protection under the copyright laws of the United States andof other countries. The owner of the copyright rights has no objectionto the facsimile reproduction by anyone of the patent document or thepatent disclosure, as it appears in the United States Patent andTrademark Office publicly available file or records, but otherwisereserves all copyright rights whatsoever. The copyright owner does nothereby waive any of its rights to have this patent document maintainedin secrecy, including without limitation its rights pursuant to 37C.F.R. §1.14.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention pertains to software-based fault tolerant computersystems, computer networks, telecommunications systems, embeddedcomputer systems, wireless devices such as cell phones and PDAs, andmore particularly to methods, systems and procedures (i.e., programming)for consistent replication of application programs across two or moreservers.

2. Description of Related Art

In many environments one of the most important features is to ensurethat a running application continues to run even in the event of one ormore system or software faults. Mission critical systems intelecommunications, military, financial and embedded applications mustcontinue to provide their service even in the event of hardware orsoftware faults. The auto-pilot on an airplane is designed to continueto operate even if some of the computer and instrumentation is damaged;the 911 emergency phone system is designed to operate even if the mainphone system if severely damaged, and stock exchanges deploy softwarethat keep the exchange running even if some of the routers and serversgo down. Today, the same expectations of “fault-free” operations arebeing placed on commodity computer systems and standard applications.

Fault tolerant systems are based on the use of redundancy (replication)to mask faults. For hardware fault tolerance, servers, networking orsubsystems are replicated. For application fault tolerance, theapplications are replicated. Faults on the primary system or applicationare masked by having the backup system or application (the replica) takeover and continue to provide the service. The take-over after a fault atthe primary system is delicate and often very system or applicationspecific.

Several approaches have been developed addressing the fundamentalproblem of providing fault tolerance. Tandem Computers is an example ofa computer system with custom hardware, custom operating system andcustom applications, offering transaction-level fault tolerance. In thisclosed environment, with custom applications, operating system andhardware, a fault on the primary system can be masked down to thetransaction boundary and the backup system and application take overseamlessly. The fault-detection and failover is performed in real-time.

In many telecommunication systems fault tolerance is built in. Redundantline cards are provided within the switch chassis, and if one line cardgoes down, the switching fabric automatically re-routes traffic and liveconnections to a backup line card. As with the Tandem systems, manytelecommunications systems are essentially closed systems with customhardware, custom operating systems and custom applications. The faultdetection and failover is performed in real-time.

In enterprise software systems the general approach taken is thecombined use of databases and high availability. By custom programmingthe applications with hooks for high-availability it is generallypossible to detect and recovery from many, but not all, types of faults.In enterprise systems, it is typically considered “good enough” torecover the application's transactional state, and there are often nohard requirements that the recovery be performed in real-time. Ingeneral, rebuilding the transactional state for an application servercan take as much as 30 minutes or longer. During this time, theapplication services, an e-commerce website for instance, is unavailableand cannot service customers. The very slow fault recovery can to someextent be alleviated by extensive use of clustering and highlycustomized applications, as evidenced by Amazon.com and ebay.com, butthat is generally not a viable choice for most deployments.

In U.S. Pat. No. 7,228,452 Moser et al teach “transparent consistentsemi-active and passive replication of multithreaded applicationprograms”. Moser et al disclose a technique to replicate runningapplications across two or more servers. The teachings are limited tosingle process applications and only address replica consistency as itrelated to mutex operations and multi-threading. Moser's invention doesnot require any modification to the applications and work on commodityoperating systems and hardware. Moser is incorporated herein in itsentirety by reference.

The present invention builds on the teachings in U.S. patent applicationSer. No. 12/851,706 titled SYSTEM AND METHOD FOR TRANSPARENT CONSISTENTAPPLICATION-REPLICATION OF MULTI-PROCESS MULTI-THREADED APPLICATIONS inwhich Havemose disclose systems and methods for transparent consistentapplication replication.

Therefore, a need exists for systems and methods for providingtransparent application-replication that address all types ofapplications, including multi-process multi-threaded application,application that use any type of locking mechanisms and application thataccess any type of external resources. Furthermore, theapplication-replication must be consistent and work on commodityoperating system, such as Windows and Linux, and commodity hardware withstandard applications.

BRIEF SUMMARY OF THE INVENTION

The present invention provides systems and methods forapplication-replication that is consistent, transparent and works oncommodity operating system and hardware. The terms“Application-replication” or “replication” are used herein to describethe mechanism by which two copies of an application are kept running invirtual lock step. The application-replication in the present inventionuses a leader-follower (primary-backup) strategy, where the (primary)application runs on the primary server and the backup application (alsocalled the “replica”) runs on a backup server. While it's possible torun the primary application and the backup application on the samephysical server, the primary and backup are generally depicted asseparate servers.

The primary application runs at full speed without waiting for thebackup, and a messaging system, a key component of the presentinvention, keeps the backup application in virtual lock step with theprimary.

A replication strategy is said to achieve “replica consistency” or be“consistent” if the strategy guarantees that the primary and backupapplication produce the same results in the same order. Replicaconsistency is critical with multi-process applications where thevarious parts of the application execute independently of each other.Replica consistency is a key element of the present invention and isexplained in further detail below.

The term “virtual lock-step” is used to describe that the applicationand the application's replica produce the same results in the sameorder, but not necessarily at the same time; the backup may be behind.

The terms “primary” and “primary application” are used interchangeablyto designate the primary application running on the primary host. Thehost on which the primary application is running is referred to as the“primary server”, “primary host” or simply the “host” when the contextis clear. The term “on the primary” is used to designate an operation oractivity related to the primary application on the primary server.

Similarly, the terms “backup” and “backup application” are usedinterchangeably to designate a backup application running on a backuphost. The host on which the backup application is running is referred toas a “backup server”, a “backup host” or simply a “host” when thecontext is clear. The terms “on the backup” or “on a backup” are usedinterchangeably to designate an operation or activity related to abackup application on a backup server.

The following terms are used throughout the disclosures:

The terms “Windows” and “Microsoft Windows” is utilized hereininterchangeably to designate any and all versions of the MicrosoftWindows operating systems. By example, and not limitation, this includesWindows XP, Windows Server 2003, Windows NT, Windows Vista, WindowsServer 2008, Windows 7, Windows Mobile, and Windows Embedded.

The terms “Linux” and “UNIX” is utilized herein to designate any and allvariants of Linux and UNIX. By example, and not limitation, thisincludes RedHat Linux, Suse Linux, Ubuntu Linux, HPUX (HP UNIX), andSolaris (Sun UNIX).

The term “node” and “host” are utilized herein interchangeably todesignate one or more processors running a single instance of anoperating system. A virtual machine, such as VMWare, KVM, or XEN VMinstance, is also considered a “node”. Using VM technology, it ispossible to have multiple nodes on one physical server.

The terms “application” is utilized to designate a grouping of one ormore processes, where each process can consist of one or more threads.Operating systems generally launch an application by creating theapplication's initial process and letting that initial processrun/execute. In the following teachings we often identify theapplication at launch time with that initial process.

The term “application group” is utilized to designate a grouping of oneor more applications.

In the following we use commonly known terms including but not limitedto “client”, “server”, “API”, “Java”, “process”, “process ID (PID)”“thread”, “thread ID (TID)”, “thread local storage (TLS)”, “instructionpointer”, “stack”, “kernel”, “kernel module”, “loadable kernel module”,“heap”, “stack”, “files”, “disk”, “CPU”, “CPU registers”, “storage”,“memory”, “memory segments”, “address space”, “semaphore”, “loader”,“system loader”, “system path”, “sockets”, “TCP/IP”, “http”, “ftp”,“Inter-process communication (IPC), “Asynchronous Procedure Calls (APC),“POSIX”, “certificate”, “certificate authority”, “Secure Socket Layer”,“SSL”, MD-5”, “MD-6”, “Message Digest”, “SHA”, “Secure Hash Algorithm”,“NSA”, “NIST”, “private key”, “public key”, “key pair”, and “hashcollision”, and “signal”. These terms are well known in the art and thuswill not be described in detail herein.

The term “transport” is utilized to designate the connection, mechanismand/or protocols used for communicating across the distributedapplication. Examples of transport include TCP/IP, Message PassingInterface (MPI), Myrinet, Fibre Channel, ATM, shared memory, DMA, RDMA,system buses, and custom backplanes. In the following, the term“transport driver” is utilized to designate the implementation of thetransport. By way of example, the transport driver for TCP/IP would bethe local TCP/IP stack running on the host.

The term “interception” is used to designate the mechanism by which anapplication re-directs a system call or library call to a newimplementation. On Linux and other UNIX variants interception isgenerally achieved by a combination of LD_PRELOAD, wrapper functions,identically named functions resolved earlier in the load process, andchanges to the kernel sys_call_table. On Windows, interception can beachieved by modifying a process' Import Address Table and creatingTrampoline functions, as documented by “Detours: Binary Interception ofWin32 Functions” by Galen Hunt and Doug Brubacher, Microsoft ResearchJuly 1999”. Throughout the rest of this document we use the terminterception to designate the functionality across all operatingsystems.

The term “transparent” is used herein to designate that no modificationto the application is required. In other words, the present inventionworks directly on the application binary without needing any applicationcustomization, source code modifications, recompilation, re-linking,special installation, custom agents, or other extensions.

To avoid simultaneous use of shared resources in multi-threadedmulti-process applications locking is used. Several techniques andsoftware constructs exists to arbitrate access to resources. Examplesinclude, but are not limited to, mutexes, semaphores, futexes, criticalsections and monitors. All serve similar purposes and often vary littlefrom one implementation and operating system to another. In thefollowing, the term “Lock” is used to designate any and all such lockingmechanism. Properly written multi-process and multi-threaded applicationuse locking to arbitrate access to shared resources

The context of the present invention is an application on the primaryserver (primary application or the primary) and one or more backupapplications on backup servers (also called the replicas or backups).While any number of backup-servers with backup applications is supportedthe disclosures generally describe the scenario with one backup. As isobvious to anyone skilled in the art this is done without loss ofgenerality.

As part of loading the primary application interceptors are installed.The interceptors monitor the primary applications activities and sendsmessages to the backup. The backup uses said messages to enforce theprimary's execution order on the backup thereby ensuring replicaconsistency.

A key element of the present invention is thus the combined use ofinterceptors and a messaging subsystem to provide replicate consistency.

Another aspect of the present invention is that the replica consistencyis achieved without requiring any application modifications. Theapplication replication is provided as a system service and is fullytransparent to the application.

Another aspect of the present invention is the use of sequence numberingto capture the execution stream of for multi process and multi threadedapplications. Yet another aspect is the use of the sequence numbers onthe backup to enforce execution that is in virtual synchrony with theprimary.

A further aspect of the present invention is that it can be provided oncommodity operating systems such as Linux and Windows, and on commodityhardware such as Intel, AMD, SPARC and MIPS. The present invention thusworks on commodity operating systems, commodity hardware with standard(off the shelf) software without needing any further modifications.

One example embodiment of the present invention includes a system forproviding replica consistency between a primary application and one ormore backup applications, the system including one or more memorylocations configured to store the primary application executing for ahost with a host operating system. The system also includes aninterception layer for the primary application intercepting calls to thehost operating system and to shared libraries and generating replicationmessages based on said intercepted calls, a messaging engine for theprimary application sending said replication messages to the one or morebackup applications, and one or more additional memory locations areconfigured to store the one or more backup applications executing forone or more hosts each with a corresponding host operating system. Thesystem further includes one or more additional messaging engines foreach backup application receiving said replication messages from theprimary application, and backup interception layers corresponding toeach backup intercepting call to the operating system and sharedlibraries. The ordering information is retrieved from the one or moreadditional messaging engines for each backup application, and eachreplication message contains at least the process ID, thread ID and asequence number, and replica consistency is provided by imposing thesame call ordering on backup applications as on the primary application.

Further aspects of the invention will be brought out in the followingportions of the specification, wherein the detailed description is forthe purpose of fully disclosing preferred embodiments of the inventionwithout placing limitations thereon.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

The invention will be more fully understood by reference to thefollowing drawings which are for illustrative purposes only:

FIG. 1 is a block diagram of the core system architecture for bothprimary and backups

FIG. 2 is a block diagram illustrating a pair of primary and backup

FIG. 3 is a block diagram illustrating Interception

FIG. 4 is a block diagram illustrating creation of replication messagesby the primary

FIG. 5 is a block diagram illustrating the primary's messaging engine

FIG. 6 is a block diagram illustrating a backup's messaging engine

FIG. 7 is a block diagram illustrating handling of PROCESS messages

FIG. 8 is a block diagram illustrating a backup's processing replicationmessages

FIG. 9 is a block diagram illustrating I/O write processing

FIG. 10 is a block diagram illustrating various deployment scenarios.

DETAILED DESCRIPTION OF THE INVENTION

Referring more specifically to the drawings, for illustrative purposesthe present invention will be disclosed in relation to FIG. 1 throughFIG. 10 It will be appreciated that the system and apparatus of theinvention may vary as to configuration and as to details of theconstituent components, and that the method may vary as to the specificsteps and sequence, without departing from the basic concepts asdisclosed herein.

0. Introduction

The context in which this invention is disclosed is an applicationrunning on a primary server and one or more replicated instances of theapplication running on one or more backup servers. Without affecting thegeneral case of multiple replicated backup applications, the followingdisclosures often depict and describe just one backup. Multiple backupsare handled in a similar manner.

Similarly, the disclosures describe one primary application. Multipleapplications are handled in a similar manner. Likewise, the disclosuresgenerally describe applications with one or two processes; any number ofprocesses is handled in a similar manner. Finally, the disclosuresgenerally describe one or two threads per process; any number of threadsis handled in a similar manner

1. Overview

FIG. 1 illustrates by way of example embodiment 10 the overall structureof the present invention for both primary and backups. The followingbrief overview illustrates the high-level relationship between thevarious components; further details on the inner workings andinterdependencies are provided in the following sections. FIG. 1.Illustrates by way of example embodiment a primary and backup server 12with an application 16 loaded into system memory 14. The application 16is comprised of two processes; process A 18 and process B 20. Each ofthe two processes has two running threads. Process A contains thread T022 and thread T1 24, while process B is contains thread T3 26 and threadT4 28. An interception layer (IL) 30,32 is interposed between eachapplication process and the Messaging Engine (ME) 34, the systemlibraries 36 and operating system 38. Process A's interception Layer 30and Process B's interception Layer 32 use the shared messaging engine(ME) 34 to send and receive messages used to enforce replicataconsistency.

System resources, such as CPUs 46, I/O devices 44, Network interfaces 42and storage 40 are accessed using the operating system 38. Devicesaccessing remote resources use some form of transport network 48. By wayof example, system networking 42 may use TCP/IP over Ethernet transport,Storage 40 may use Fibre Channel or Ethernet transport, and I/O may useUSB.

In the preferred embodiment storage 40 is external and accessible byboth primary and backups.

The architecture for the primary and backups are identical. At thefunctional level, the Messaging Engine 34 generally is sending outreplication messages on the primary, while the ME 34 on the backup isreceiving and processing replication messages sent by the primary.

FIG. 2 illustrates by way of example embodiment 60 a primary server 62and its corresponding backup server 82 working as a pair of primary andbackup. The primary application 64 is comprised of two processes;process A 66 and process B 68, each with two running threads. ProcessA's interception layer 70 and the Messaging Engine 74 are interposedbetween process A 66 and the operating system and libraries 76.Likewise, Process B's interception layer 72 and the Messaging Engine 74are interposed between process B 68 and the operating system andlibraries 76.

Using a similar architecture, the backup server 82 contains the backupapplication (the replica) 84 comprised of process A 86 and process B 88each with two threads. The Interception Layers IL 90 for process A andIL 92 for process B are interposed together with the Messaging Engine 94between the two processes and the system libraries and operating system96.

As illustrated on both FIG. 1 and FIG. 2 there is one Messaging Engineper application. If an application contains multiple processes, theapplication processes share one message engine.

2. Interception

Interception is used to intercept all events, library calls and lockingcalls that affect replica consistency. FIG. 3 illustrates by way ofexample embodiment 100, the core interception architecture for anapplication with two processes. Details on the Messaging Engine and itsarchitecture are given below. Process A 102 with interception layer 106,and process B 112 with interception layer 116. By way of example,ifunc1( ) and ifunc2( ) are subject to interception. When process A 102reaches ifunc1( ) it is intercepted 108 and the call redirected to theinterception layer 106. The interception layers processes the ifunc1( )calls as follows (in pseudo code):

Call ifunc1( ) and store return values

Collect ProcessID and ThreadID for ifunc1( )

Call Message Engine 122 with (ProcessID,ThreadID) identifiers and anydata from ifunc1( ) as necessary

Return to caller 110

Upon returning to the caller 110 Process A resumes execution as ififunc1( ) had not been intercepted.

The interception mechanism is identical for process B 112, where ifunc2() 114 is intercepted 118, the interception processed 116 with the samealgorithm, and then returned 120 to the caller.

In a preferred embodiment the interception layer is implemented as ashared library and pre-loaded into each application process' addressspace as part of loading the application. Shared libraries areimplemented in such as way that each instance of the interception layershare the same code, but have their own private data. In a multi-processapplication the interception layer is therefore comprised of oneinterception layer per application process, and together theprocess-level interception layers comprise the interception layer forthe entire application.

A related issue with interception is that intercepted functions may callother intercepted functions. As long as said calls are performed usingpublic intercepted names, the previous teachings fully describe theinterception. At times shared-library developers take shortcuts anddon't use the public names, but refer directly to the implementationusing a private name. In such cases, the interceptor must overlay a copyof the intercepted shared library code using fully resolved publicfunction names.

3. Replica Consistency

Even with correctly written multi-process and multi-threaded programs,there are no guarantees that the same program run multiple timesproduces the same result at each run. By way of example consider anapplication consisting of two threads. The program contains one globalvariable, one global lock, and two threads to operate on the globalvariable. In pseudo code:

main ( )

{

int globalInt=0;

Lock globalLock=new Lock ( );

Start thread1;

Start thread2;

Print(“Final value=”+globalInt);

}

private thread1( )

{

for(int i=0; i<10; i++)

}

-   -   globalLock.lock ( );    -   globalInt=globalInt+1;    -   globalLock.unlock( );    -   sleep(random( ));

}

}

private thread2( )

{

for(int i=0; i<10; i++)

}

-   -   globalLock.lock( );    -   globalInt=globalInt*2;    -   globalLock.unlock( );    -   sleep(random( ));

}

}

Thread 1 repeats the core loop 10 times and each time first locks theglobal lock to ensure atomic access to globalInt, increments globalIntby one, frees the lock and waits a random amount of time. Thread2 hasthe same structure except it multiplies globalInt by 2.

Depending on how long each thread sleeps each time they reach sleep( )thread1 and thread2 will execute their locks in different orders andthus globalInt is not guaranteed to be the same at the end of separateruns

To ensure replica consistency, the present invention enforces anordering on events, so that the primary and backup produces the sameresults. Specifically, if the application runs on the primary andproduces a final value of 10, so will the backup. If next time theprimary produces the final value of 10240, so will the backup.

While the use of sleep( ) highlighted the consistency problem, evenwithout sleep( ) different runs would produce different final results.The reason is that the operating system schedules Thread 1 and Thread 2based on a wide range of factors, and likely will make differentscheduling decisions from run to run.

4. Generating Unique Global IDs

The present invention utilizes global IDs in several places. A “globalID” is a 64 bit integer that is guaranteed to be unique within thecontext of an application. When a new global ID is created it isguaranteed to be one larger than the most recently generated global ID.Global IDs are used as counters for replication messages. Global IDsstart at zero upon initialization and continue to increase as moreglobal IDs are requested. 64 bits ensures that integer wrap-around isnot a practical concern. In an alternate embodiment global IDs areimplemented as arbitrary precision integers, which can hold any sizeinteger and never wrap.

In a preferred embodiment generation of global IDs are provided in ashared library. On some operating systems, shared libraries can havevariables, called static library variables, or global library variables,that are shared across all instances of the shared library. For suchoperating system, the preferred implementation uses such global libraryvariables to implement the global IDs. In pseudo code the implementationis, where “m_GlobalID” is the global shared variable:

static Int64 m_GlobalID=0;

Lock m_GlobalIDLock=new Lock( );

static int64 createGlobalID( )

{

-   -   Int64 id=m_GlobalID;    -   m_GlobalIDLock.lock( );    -   m_GlobalID=m_GlobalID+1;    -   id=m_GlobalID;    -   m_GlobalLock.unlock( );    -   return id;

}

Alternatively, if the operating system doesn't support global variableswithin shared libraries, the same functionality can be implemented usingshared memory, using, by way of example, the POSIX shared memorysubsystem found on modern operating system. In stead of using a staticInt64 to hold the M_GlobalID, the M_GlobalID is placed in a shmemsegment shared among all instances of the shared library and lockedusing a named semaphore This alternate technique is substantiallyidentical to the algorithm above other than the use of shared memory instead of library static variable

In a preferred implementation the global ID functionality is built intoto the Messaging Engine shared library. In an alternate implementation,the global ID functionality is provided in a separate shared library. Inthe following disclosures the global ID functionality is depicted asbeing provided by the Messaging Engine shared library, per the preferredimplantation.

5. Identifying Resources

As a thread executes it proceeds along a unique path. Generally a threadruns within the context of a process. The process has a uniqueidentifier, called the process ID or PID, and each thread has a uniqueidentifier called the thread ID or TID. In some operating systems threadIDs are globally unique, in others unique within the context of itsparent process. The combination of PID and TID uniquely identifies athread and process pair independently of whether TIDs are globally orprocess unique. On many operating systems the PID is determined by thegetpid( ) or GetProcessId( ) functions, while the TID is determined bythe gettid( ) or GetThreadId( ) functions. Other operating systems offersimilar functionality.

As an application is loaded control is first transferred from the loaderto the applications init( ) method. Generally, init( ) is provided aspart of the standard system libraries but custom init( ) may beprovided. Init( ) ends by calling the main application entry point,generally called main( ). As main( ) starts executing it does so as oneprocess with a single thread. The teachings of the present inventionfollow this model where each process automatically is created with onethread, where said thread is executing the initial program code. Thereare operating systems where every thread must be createdprogrammatically and where no initial thread is attached to a process.The present invention supports adding threads to a running process atany time, and it's thus apparent to anyone skilled in the art that thefollowing disclosures easily adapt to the case where a thread needs tobe programmatically added following process creation.

In the preferred embodiment, the present invention supplies a custominit( ) wherein all interceptors are loaded. This ensures that allresources, including threads and processes, can be intercepted and thatthe interceptors are installed before the application's main( ) iscalled.

The process and thread interceptors intercept all process and threadcreation, termination and exits. As the primary application executes anduses threads and processes, said events are communicated usingReplication Messages (described below) to the backup providing thenecessary information for the backup to rebuild the process and threadhierarchy and match it against incoming replication messages from theprimary.

By way of example, as init( ) calls main( ), the programs consists ofone process with one thread. Prior to calling main( ) a specialinitialization replication message (called PROCESS_INIT) with theinitial process ID and thread ID is sent to the backups. When a newprocess is created the new process ID together with its initial threadID are sent to the backup in a replication message (PROCESS_CREATE).Whenever a new thread is created, a replication message with the processID and new thread ID are sent to the backup (THREAD_CREATE). Likewise,whenever a process or thread terminates a replication message with theterminating process and thread is sent to the backups. The backup canthus build a representation of the process and thread hierarchy on theprimary and use that to map incoming replication messages against thebackup's own process and thread hierarchy.

To ensure replica consistency, access to all resources is interceptedand tagged, so that the identical access sequence can be imposed on thereplica. The first set of interceptors intercept all process and threadcreation and termination calls. Tracking the process and threadhierarchy on the primary enables recreation of the hierarchy on thereplica. The process and thread <PID,TID> pair is attached to allresource access performed on process PID and thread TID and provides thetagging necessary to associate resource interceptors on the backup withthe corresponding process and thread on the primary

As a thread executes it does so sequentially. While a multi processand/or multi threaded application may contain many simultaneousexecuting threads and processes, each thread is performing its workserially. By way of example consider the following pseudo code:

FILE *fp=fopen(“/home/user/newfile.txt”, “w”)

if (fp !=null)

-   -   fwrite(pStr,1, strlen(pStr),fp);

fclose(fp)

The thread first opens the file using fopen( ), then writes to the fileswith fwrite( ), and finally closes the file with fclose( ). The programwill not, by way of example, first call fwrite( ), then fclose( ), andfinally fopen( ). The instruction sequence, as it relates to theresource FILE *fp, is guaranteed to be sequential as programmed in theexample code. Compilers may rearrange some of the compiled code as partof code generation and optimization, but it will always leave theresource access ordering as specified in the source code. If thecompiler re-arranges other aspects of the code execution, the samerearranged order would be in place on the backup, and such compileroptimization thus have no effect on the teachings of the presentinvention.

By way of example, this means that a thread on the primary and thebackup both would first call fopen( ), then fwrite( ) and finallyfclose( ). The present invention uses this implicit ordering to mapreplication messages against the right methods. By way of continuedexample, the backup would first, as this is how the program executes,request the replication message for fopen( ) then for fwrite( ) andfinally for fclose( ), and thus automatically match the ordering ofReplication Messages generated by the primary as far as the resourceFILE *fp is concerned.

If, by way of example, a thread uses two resources the same teachingsapply. While the compiler may have rearranged the relative order of thetwo resources, said reordering would be identical on primary and backupsand thus not affect any difference in execution on the primary and thebackups.

If by way of example, an execution environment such as Java or .NET isused, said execution environment is included as part of the applicationas the execution environment affects and controls execution.

There is thus no need to assign any resource identifiers to resources inorder to match resource on the primary with the resource on the backup.The execution context itself suffices to identify a resource and its usewithin the context of a thread and process. By way of example, thecreation of a resource by a process and thread is used directly to matchit to the corresponding process and thread on the backups. The matchingon the backups is explained in detailed below.

By way of example consider a process with two threads. The two threadsaccess a shared lock and arbitrate for access using the lock( ) andunlock( ) methods. In pseudo code

Lock globalLock=null;

private thread1( )

{

globalLock=new Lock ( ); // create

globalLock.lock( );

// do thread 1 work

globalLock.unlock( );

}

}

private thread2( )

{

globalLock.lock( );

// do thread 2 work

globalLock.unlock( );

}

}

FIG. 4 illustrates by way of example embodiment 140, the interception ofLock objects in a scenario with two threads and the creation of<PID,TID> pairs. A process is comprised of two threads, Thread-0 142 andThread-1 144. The resource interceptor 146 intercepts access to theunderlying Lock resource 148. First Thread-0 142 creates 150 the lock.The create( ) call is intercepted 152 by the resource interceptor 146.First the actual resource create( ) 154 call is performed and thereturning value stored. A replication message with the pair <PID,TID> iscreated and sent 156 to the Message Engine 141 for transmittal to thebackup. Finally the creation call return 158 the results of the resourcecreate( ) call. Later the Thread-0 142 calls the lock( ) method 160 onthe Lock object. The lock( ) is intercepted 162, and initially forwardedto the lock( ) call within the Lock object 164. The lock is returned tothe interceptor 162, and a replication message with <PID,TID> is createdand sent to the Messaging Engine. The lock is returned 168 to thread-0.At this point thread-0 has acquired the Lock and no other threads arecan acquire it while the Lock is held by thread-0.

Later thread-1 144 calls the lock( ) method 172 on the Lock object. Thelock( ) is intercepted 172 and initially is forwarded to the lock( )call within the Lock object 174. The lock( ) 174 blocks as the lock isalready acquired by Thread-0 and the call does not return to theinterceptor and thread-1 144.

Later thread-0 142 calls the unlock( ) method 180 on the Lock object.The unlock( ) is intercepted 182 and forwarded to the Lock object 184.The Lock object processes the unlock( ) 184 and returns to theinterceptor 182. A replication message with <PID,TID> is created andsent to the Message Engine 141. The unlock( ) call returns 188.

Thread-2 can now acquire the lock 174 and the lock( ) call return 190 tothe interceptor 192 where a replication message with the <PID,TID> pairis constructed and sent to the Messaging engine.

5.1 Resource Types

The present invention breaks resources down into distinct categories andhandles each separately:

1. Processes and threads and their methods: processes and threadsmethods are intercepted and used to build a mapping between processesand threads on the primary and backup.

2. Locks and their methods: Locks are intercepted and used to enforcereplica consistency relative to locks and their use

3. I/O Resources and their methods: I/O (Input/Output) resources areresources writing data to locations outside the application or readingexternal data into the application. I/O Resource methods are interceptedand additional replication messages corresponding are added. Example I/Oresource methods that write data include, but are not limited to, write() for files, srand(n) where the srand(s) sets the seed value for arandom number generator, and sendmsg( ) from the sockets library. Allthree examples write data to a location outside the application proper.Example I/O resource methods that read data include, but are not limitedto, read( ) for files, rand( ) to generate a random number,gettimeofday( ), and readmsg( ) from the sockets library. All fourexamples reads or generates external data and delivers it into theapplication proper.

4. Other and Special Cases

All classes of resources are included in the teachings of the presentinvention. I/O Resources are the most general type of resource andprovide additional information in the replication messages. Any resourcenot included in the first two groups is treated as an I/O resource eventhough the functionality may not be I/O related.

6. Replication Messages

Replication Messages use the following Layout

METHOD_ID, Sn, PID,TID, DATA

Where “METHOD_ID” is one of a few pre-defined method IDs, “Sn” is thereplications sequence number, “PID” is the process ID, “TID” is thethread ID, and “DATA” is an additional field that in some case carryextra information.

The sequence number is a global ID generated and added by the MessagingEngine to every replication message. Each new sequence number is exactlyone larger than the previous sequence number, and is used on the backupto impose the same ordering as on the primary.

Example METHOD_IDs include

PROCESS_INIT used to initialize the process and thread hierarchy

PROCESS_CREATE used to designate the creation of a new process

THREAD_CREATE used to designate the creation of a new thread

PROCESS_EXIT used to designate the termination of a process andassociated threads

THREAD_EXIT used to designate the termination of a thread

METHOD_NONE used to designate that no special method ID is required

In the preferred embodiment, Method IDs are integers and predefined. Inthe preferred embodiment METHOD_NONE is defined as zero or null,indicating that the method is implicitly provided via the sequentialexecution of the thread.

Every time a resource is created, accessed, or used a replicationmessage is created on the primary and sent via the messaging engine tothe backup. The replication message contains the process and threadwhere the resource was accessed and a sequence number ensuring strictordering of events. To distinguish the replication messages from thesurrounding text it is at times enclosed in “<” and “>”. Those specialcharacters are not part of the replication messages and are usedentirely for clarify of presentation.

As disclosed previously, the implicit ordering of execution within athread is used to order resource access and the present invention thusdoes not need to specify the nature of the intercepted method; theinterception ordering is identical on the backups and the correspondingprimary. Therefore, most replication message has a METHOD_ID ofMETHOD_NONE as the primary and backup process the resource requests inthe same sequential order and need no further data to identify resourceand interception.

Continuing the example embodiment referred to in FIG. 4, the messagesgenerated by the Resource Interceptor, has a process ID of ‘P’, threadID of T0 for Thread-0 142, and thread ID of T1 for Thread-1 144. By wayof example we identify the sequence numbers as S0, S1, S2 etc.

METHOD_NONE, S0, P, T0 // new Lock( ), Thread 0 METHOD_NONE, S1, P, T0// lock( ), Thread 0 METHOD_NONE, S2, P, T0 // unlock( ), Thread 0METHOD_NONE, S3, P, T1 // lock( ), Thread 1

Where everything after and including “//” are comments included only forclarity of presentation

The messages and the ordering implied by the ever increasing sequencenumbers S0, S1, S2 and S3 describe the ordering, use and access ofshared resources. If a library method exists in two variants withdifferent signatures, each method is intercepted and generates its ownmessage, if Lock.lock( ) had two different signatures, and thread-1 144used the alternate method, the replication messages would look the same,as the backup automatically would be executing the alternate lockimplementation on thread-1 as well.

METHOD_NONE, S0, P, T0 METHOD_NONE, S1, P, T0 METHOD_NONE, S2, P, T0METHOD_NONE, S3, P, T1 // second lock( ) signature

If the operating system provided two methods to create new processes,there would be both a PROCESS_CREATE and PROCESS_CREATE2, wherePROCESS_CREATE2 designates the alternate method to create processes.

As disclosed above, process and threads require special considerationand have their own replication messages. Upon creating a new process aspecial PROCESS_CREATE replication message is sent to the backups. ThePROCESS_CREATE identifies the new process ID, its corresponding threadID and its parent process. The parent process ID is encoded in the DATAfield. Upon creating a new thread, the new thread ID, its correspondingprocess' PID, and the threads parent thread ID encoded in the DATAfield, is sent within a THREAD_CREATE replication message to thebackups. Depending on when the operating system schedules the newprocess and thread they will get to run either before or after theparent process and thread. On the backups, the messaging engine may thusreceive messages from the newly created process or thread beforereceiving the PROCESS_CREATE or THREAD_CREATE replication messages, oralternatively receive requests for PROCESS_CREATE or THREAD_CREATEmessages before the messages from the primary have arrived. Themessaging engine on the backups automatically suspends requests from thenew processes and threads until the mapping of process and thread IDhave been established as disclosed later.

By way of example, the process replication messages corresponding to aprogram starting, creating one new process called P1, then terminatingP1, are:

PROCESS_INIT, S0, P0, T0 PROCESS_CREATE, S1, P1, T1, P0 PROCESS_EXIT,S2, P1, T1

Where S0, S1 and S2 are the sequence numbers, P0 the process ID of theinitial process, T0 the thread ID of the thread for P0. P1 is theprocess ID of the created process while T1 is the thread ID of the firstthread in P1. The parent process's process IDs is provided as DATA forPROCESS_CREATE. PROCESS_INIT is the special previously disclosedinitialization message sent just prior to entering main( ).

At times a replication message optionally includes additional data. Thedata is appended in the DATA block and transmitted along with the corereplication message. The DATA block contains the DATA identifier, a 64bit long identifying the length of the data block, and the data itself.By way of example, a replication message for a fwrite( ) operation maylook like METHOD_NONE S0, P0, T0, {DATA,len,datablock}

DATA blocks are used primarily to send complex data such as data writtento files, results of operations and success/failure of operations. TheDATA blocks are primarily used with I/O Resources. The curly brackets“{” and “}” are not part of the message, they are used here for clarityof presentation. The DATA block is also used by PROCESS_CREATE todesignate the parent process's PID.

7. Message Engine

FIG. 5 illustrates by way of example embodiment 200, the structure ofthe Message Engine 201 on the primary. The base replication message issent to the Message Engine 206 where it's received 212. A sequencenumber is requested 214 from the Sequence Number generator 210, andadded to the message. The message is ready for transmission 218 to thebackup over the network 219.

In the preferred embodiment Sequence Numbers are generated with thepreferred Global ID embodiment disclosed above.

The message engine on the backup receives all the replication messagesand sorts them by sequence number. The sequence number in thereplication message identifies the order in which events previously tookplace on the primary, and therefore must be imposed on the backup duringexecution. As disclosed above and illustrated on the example embodimenton FIG. 4, the resource interceptor relies on the underlying operatingsystem and system libraries to supply the native resource access andlocking, and then tags on the process, thread, and sequence numbers toidentify the context and relative order.

FIG. 6 illustrates by way of example embodiment 220 the Message Engine221 on a backup. Replication messages are received 224 over the network222. Depending on underlying transport, Replication Messages may arriveout of order: In a preferred embodiment using TCP, TCP ensures messageordering. In an alternate preferred embodiment using UDP, there is noguarantee that messages arrive in the same order they were sent. Ingeneral, Replication Messages may thus in general arrive out of orderand are therefore sorted 226 by sequence number. A sorted list of newmessages 228 is maintained by the present invention within the MessageEngine 221 on the backups. By way of example, a message with sequencenumber 100 is sent, followed by a message with sequence number 101, theymay arrive out-of-order on the backup, so that the message with sequencenumber 101 arrives prior to the replication message with sequence number100. The sorting step 226 ensures that the oldest replication messagewith lowest sequence number is kept at the top, while later messages areplaced in their sorted order later in the list 228.

When the resource interceptors on the backup requests a replicationmessage 232, the request is processed by the request module 230. Inorder to deliver a replication message to an interceptor two tests mustbe passed:

Test 1—Sequence Number:

The request module 230 compares the sequence number at the top of thesorted list of replication messages 228 with the sequence number of themost recent message 236. If top of the list 228 has a sequence number ofexactly one more than the most recent sequence number 236 thetop-message is a candidate for delivery to the calling interceptor 232,234. If the top-message sequence number is more than one larger than thelast sequence number 236, one or more replication messages are missing,and the request module 230 pauses pending the arrival of the delayedmessage.

By way of example, and in continuation of the example above, if the lastsequence number is 99, and the message with sequence number 101 hasarrived, while the message with sequence number 100 has not arrived, therequest module 230 waits until the message with sequence number 100 hasbeen received and placed at the top of the sorted list. Upon arrival ofthe replication message with sequence number 100, said message is now acandidate for delivery to the calling interceptor 232, 234 provided thesecond test passes.

Test 2—METHOD_ID, Process ID and Thread ID:

The caller 232 supplies METHOD_ID, PID, TID and parent PID, whenrequesting a replication message. This means that the callinginterceptor is requesting the oldest replication message of typeMETHOD_ID with process ID of PID and thread ID of TID.

When METHOD_ID is METHOD_NONE the requested method is implicit in theserial execution of the thread and it suffice to compare process ID andthread ID. By way of example, to retrieve the replication message forprocess B-P0 and Thread B-T1, the interceptor would supply parameters ofB-P0 and B-T1 which are the process ID and thread ID of the interceptorand calling application on the backup. The replication messages containPIDs and TIDs from the primary. As the backup executes, each process andthread generally have different IDs than the corresponding threads onthe primary. The present invention maintains a mapping 233 between the<PID,TID> pairs on the primary and the corresponding pairs on the backup<B-PID, B-TID>. Detailed teachings on creation and management of saidmapping is given in section 8. The interceptors, when requesting areplication message 232, provide B-P0 and B-T1 as those are its localprocess and thread IDs. The replication request module 230 thentranslates the local process and thread IDs, using the PID-TID mapping233 into the primary <PID,TID> and uses said primary <PID,TID> in theprocess and thread ID comparisons described. If the replication messageat the top of the list 228 has a <PID,TID> that matches the translated<B-T0,B-T1> there is a match and test is successful.

If the METHOD_ID provided by the calling interceptor 232 is differentfrom METHOD_NONE, special processing is required. Replication messagesrelated to process and threads have their own METHOD_IDs and are thushandled with special processing. By way of example, to retrieve thereplication message for PROCESS_CREATE, the calling interceptor suppliesparameters of PROCESS_CREATE, B-P1,B-T1,B-P0, where B-P1 is the newlycreated process with initial thread of B-T1, and B-P0 is its parentprocess. When requesting the replication message for PROCESS_CREATE onlythe parent process B-P0 is already mapped in the translations 233. Foran incoming PROCESS_CREATE message with parent process P0, thecorresponding B-P0 can be found in the mappings 233 as the processpreviously was mapped. If a process ID match is found for the parentprocesses, the “new process”<P1,T1> pair from the replication message ismapped against the <B-P1,B-T1> pair supplied in the interceptor andadded to the mappings 233 and the test is successful.

Similarly teachings apply for THREAD_CREATE, where the parent's threadID and the process ID are the two known quantities. Creation andmaintenance of the mappings 233 is explained in further detail insection 8.

If both tests are satisfied, the top replication message is removed fromthe list and returned 234 to the calling interceptor and the lastsequence number 236 updated to the sequence number of the just-returnedmessage 234.

The combined use of sequence numbers, which ensure that only the oldestmessage is delivered, combined with the full calling context of P0 andT1 enable the Replication Request Module 230 to only return replicationmessages that are designated for the particular thread and process. If athread requests a replication message and the particular message isn'tat the top of the list, the thread is placed in a “pending threadscallback” queue 231. As soon as the requested message is available atthe top of the message list 228, the thread is removed from the “pendingthreads callback” queue 231 and the call is returned 234. The mechanismof pausing threads where the replication messages are not available orat the top of the message list 228 is what enables the present inventionto enforce replica consistency on the backup even when processes andthreads are scheduled differently on the backup than they were on theprimary.

Further teachings on the use of replication messages by the interceptorson the backups, and the access methods are disclosed next

8. Processing Replication Messages on the Backup

The backup is launched and interceptors are installed in init( ) asdisclosed above for the primary. On the backup, however, init does notimmediately call main( ); rather it requests and waits for thePROCESS_INIT message from the primary before proceeding. Where theprimary runs unimpeded and sends replication messages when accessingresources, the backup conversely stops immediately upon entering aresource interceptor and retrieves the replication message correspondingto the particular event before proceeding.

Generally, operating systems assign different process IDs, thread IDs,resource handles etc. each time an application is run. There is thus noguarantee that a particular application always gets the same process ID.This means that the initial process on the primary and the initialprocess on the backup may have different process IDs. Likewise for allother resources. To correctly map replication messages from the primaryto interceptors on the backups a mapping of between process and threadIDs on the primary and backup is created.

As the initial process is created and just prior to calling main, anreplication message <PROCESS_INIT,S0,P0,T0> is created and sent to thebackup. On the backup, the messaging engine receives the PROCESS_INITmessage. Referring to FIG. 6 for illustrative purposes: When theinterceptor on the backup requests 232 the PROCESS_INIT it supplies itsprocess and thread IDs (B-P0, B-T0). The replication request module 230is thus able to match the <P0,T0> pair with <B-P0,B-T0> and creates anentry in the PID-TID mapping 233. Likewise, when a PROCESS_CREATE orTHREAD_CREATE message is at the top of the sorted message list 228, thereplication request module 230 creates a mapping between the newlycreated process's and/or thread's primary and backup IDs. When a processor thread terminates and sends PROCESS_EXIT or THREAD_EXIT, thereplication request module 230 similarly removes the related entry fromthe PID-TID mappings upon receiving the request 232 from theinterceptor. The Replication Request module 230 thus dynamicallymaintains mappings between <PID,TID> pairs on the primary and thecorresponding <B-PID,B-TID> on the backup.

In the preferred embodiment the messaging engine maintains the processand thread ID mappings. In an alternate embodiment the interceptorsmaintain the mappings

In the preferred embodiment, the mapping between processes and threadson the primary <Pi,Ti> and their counterparts on the backups <B-Pi,B-Ti> are maintained using a hash table, with the <Pi,Ti> pair being thekey and the pair <B-Pi,B-Ti> being the corresponding process/thread onthe backup. In an alternate embodiment a database is used to maintainthe mappings.

FIG. 7 illustrates by way of example embodiment 240 an applicationstarting as one process P0 242. The application starts and gets to init244 where interceptors are installed. Before calling main 245 thereplication message 254<PROCESS_INIT S0,P0,T0> is created and sent tothe Message engine 241. The initial process P0 contains one thread T0246. At some point during execution a second process P1 248 is created.A replication message 256 <PROCESS_CREATE,S1,P1,T3,P0> is createddesignating the process, the initial thread T3 250, and the parentprocess P0. Said message is transmitted via the Messaging Engine 241. Asecond thread T4 252 is later created within the process P1. Thecorresponding replication message <THREAD_CREATE,S2,P1,T4,T3> is created258 and transmitted via the message engine 241.

On the backup incoming replication messages are sorted by sequencenumber, and the process and thread ID mappings are created as previouslydisclosed The list of replication messages are

PROCESS_INIT S0, P0, T0, P0 PROCESS_CREATE, S1, P1, T3, P0THREAD_CREATE, S2, P1, T4, T3

On the backup, the application is started 262 and gets to init 264 whereinterceptors are installed. Where the primary sends out the PROCESS_INITmessage prior to calling main( ) the backup in stead requests thePROCESS_INIT message from the message engine 261. The message engine,delivers the message 274<PROCESS_INIT S0, P0,T0,P0> to init 264. ThePROCESS_INIT replication message allows the backup messaging engine tomap its process ID of B-P0 to P0 and B-T0 to primary thread ID T0.Henceforth, whenever a replication message with process ID of P0 isreceived, the backup maps it to the process with ID B-P0. Likewisereplication messages with thread ID of T0 are mapped to B-T0 on thebackup. The backup proceeds to main 265 and begins to execute. Laterduring the single-threaded execution of B-P0 a second process B-P1 iscreated. The “process create” is intercepted as part of the interceptorsfor processes and threads. After creating the process B-P1 268 and theinitial thread B-T3 270 the message engine is called again. The requestis for a <PROCESS_CREATE> message 276 with parent process P0. At the topof the list is <PROCESS_CREATE,S1,P1,T3,P0> which is the correctmessage, and its returned to the calling interceptor. The messagingengine can now map P1 to B-P1 and T3 to B-T3. Later during the executionof thread B-T3 a thread_create( ) is encountered. The thread is createdand a THREAD_CREATE message is requested with process ID P1 and threadID P3. At the top of the list is <THREAD_CREATE, S2,P1,T4> which is thecorrect message and its returned 278 to the interceptor. The messagingengine can now map thread ID T4 to B-T4 on the backup.

FIG. 8 illustrates by way of example embodiment 280, processing of thereplication messages on the backup generated by the embodiment of theprimary shown on FIG. 4. The replication messages generated by theprimary were disclosed above as:

METHOD_NONE, S0, P, T0 // new Lock( ), Thread 0 METHOD_NONE, S1, P, T0// lock( ), Thread 0 METHOD_NONE, S2, P, T0 // unlock( ), Thread 0METHOD_NONE, S3, P, T1 // lock( ), Thread 1

The following assumes that the process and thread mappings have beenestablished as taught above and mapping thus exists between threads andprocesses on the primary and the backup. Thread-0 282 is the thread onthe backup corresponding to thread-0 FIG. 4-142 while Thread-1 284 isthe thread on the backup corresponding to thread-1 FIG. 4-144. Theinterceptor for Lock 286 was installed during init( ), and the Lockresource is 288.

Initially, Thread-0 282 calls create( ) 290 to create the resource. Thecall is intercepted 292. The interceptor requests the replicationmessage for process P and Thread T0. The message with matching <PID,TID>is at the top of the message list in the messaging engine 281 and isreturned to the interceptor. The interceptor proceeds to call theresource create( ) 294 and returns the resource to the calling thread 0296.

By way of example, on the backup thread 2 284 is scheduled to run andthread 2 request the lock ( ) 290 prior to thread 1 requesting the lockas were the case illustrated on FIG. 4. The call is intercepted 292 andthe message for process P and thread T1 is requested. This message withmatching <PID,TID> is not at the top of the list in the messaging engine281 and thread T1 284 thus is blocked and put on the Pending ThreadsCallback list and the call not returned to the interceptor.

Thread 0 282 is then scheduled and requests a lock( ) 300 on theresource. The call is intercepted 302, and the message for process P andthread T0 is requested. The is the message with matching <PID,TID> is atthe top of the message list 281 and is thus returned to the callinginterceptor 302. The interceptor calls lock( ) in the resource 304 andreturns the lock to the called 306. After using the lock'ed objectedunlock 310 is called an intercepted 312. The replication message withmatching <PID,TID> for process P and thread T0 is requested and returnedas it's at the top of the message list 381. The interceptor 312 callsthe resource unlock( ) and the resource is unlocked.

Upon delivering the replication message corresponding to unlock( ) 310for Thread 0 to the interceptor 312 the earlier request from thread 1284 containing <P,T1> is now at the top of the list in the messagingengine 281. The message is therefore returned to the interceptor 322 andlock ( ) is called in the resource 324. If Thread 1 282 has not yetcalled unlock ( ) 314 the resource lock 324 blocks until the resource isunlocked by thread 0 282. If thread 0 has unlocked the resource 316 theresource lock 324 would immediately succeed and return the interceptor322. The lock is then returned 326 to the calling thread.

The present invention thus ensures that the lock ordering from theprimary is enforced on the backup, even if the backup requests locks ina different order. It is readily apparent to anyone skilled in the artthat the teachings extends to multiple locks, processes, threads andobjects and that the teachings thus ensures replica consistency betweenthe primary and backup.

9. I/O Resource Methods

The teachings so far have focused on processes, threads and locks. I/OResource methods may write data to locations outside the applicationproper. By way of example, the locations can be files on disk, locationsin memory belong to the operating system or system libraries, orlocations addressable over a network. The data written with writingmethods persists beyond the write operation: data is stored in files,the seed for a random number generator affects future random( ) calls,and data written to a socket is received by the another application.

9.1 I/O Resources—Writing Data

Write operations generally cannot be repeated. By way of example, ifdata is appended to a file (a write operation) appending the data asecond time produces a different file larger file with the data appendedtwice. This present invention addresses this issue by ensuring that thebackup, by way of continued example, doesn't append the data to the fileeven though the primary performed an append write operation. Writeoperations on the backup are suppressed, i.e. the interceptors capturethe results from the primary application and use those on the backup instead of performing the actual write. This aspect of the presentinvention is explained in further detailed below.

The primary application run unimpeded and performs all write operations.The replication messages corresponding to write operations are similarto the ones used for locks. However, write operations may have returnvalues indicating, by way of example, the number of bytes written, andmay modify some of the parameters passed to the method of the writeoperation. This additional information is also packed into replicationmessages and sent to the backup using the DATA field in the replicationmessages

int main(void)

{

-   -   char const *pStr=“small text”;    -   FILE *fp=fopen(“/home/user/newfile.txt”, “w”)    -   if (fp !=null)        -   fwrite(pStr,1, strlen(pStr),fp);    -   fclose(fp)

}

By way of example, the replication messages corresponding to the aboveexample are:

METHOD_NONE, S0, P, T0, {DATA, len1, data1} //fopen( ) METHOD_NONE, S1,P, T0, {DATA, len2, data2} //fwrite( ) METHOD_NONE, S2, P, T0, {DATA,len3, data3} //fclose( )

Many write operations, such as by way of example, fwrite on a FILEopened with ‘w’ are exclusive and behave like Locks: Only one thread canwrite to a particular file at any one time. The locking behavior is thusautomatically handled, as the replication messages enforce the order ofexecution as it takes place on the primary, and thus forces the backupthrough the same locking steps in the same order.

The DATA block {DATA,len1,data1} attached to the fopen( ) replicationmessage contains the return value of the fopen( ) call, which is thefile handle. The file handle (a pointer) from the primary is of nodirect use on the backup, as the backup generally creates a differentfile handle. The contents of the FILE handle, however, containsimportant internal FILE state data such as current directory, timestamps of last access, and error conditions. The FILE handle istherefore sent to the backup so the backup can extract said internalstate and set the FILE handle state on the backup to the values from theprimary. By way of example, if fopen( ) fails on the primary, it isforced to fail on the backup, if fopen( ) succeeds on the primary, itshould succeed on the backup.

The DATA block {DATA,len2,data2} attached to the fwrite( ) replicationmessage contains the size_t object with the number of objectssuccessfully written and the FILE pointer. The count is sent to thebackup in order for the backup to return the same return value as theprimary and the FILE pointer is sent so that the backup can update itslocal FILE point to have the same internal state.

For every I/O operation that writes data the return value is encoded andtransmitted in the DATA block along with the parameters. The encodingcan be as simple as an ASCII representation of the data. As long asprimary and backup agree on encoding any encoding can be used. In thepreferred embodiment the data is encoded using XML and MIME. In analternate embodiment a custom encoding is used.

The actual data written is not transmitted via a replication message.The replica already has a full running copy of the application and itcan generate the data itself if need be.

Write operations on the backup are handled much like the previousteachings with one major exception. The actual write operation issuppressed, i.e. skipped, on the backup as it generally is not valid torepeat a write operation. The results produced on the primary are“played back” on the backup. The state is adjusted based on theprimary's state as necessary.

FIG. 9 illustrates by way of example embodiment 340 the above outlinedexample of opening a file for writing, writing a string to the file,then closing the file. For clarify of presentation, the Message Engineis not shown on the diagram. FIG. 9 shows replication messages goingdirectly from the interceptor on the primary 344 to the interceptor onthe backup 346. It is however assumed that messages go through themessaging engine, are sorted by sequence number and delivered to theinterceptors on the backup as previously disclosed. Similarly, theactual I/O resource is not shown on the diagram. The resource isresponsible for writing similarly to the resource on FIG. 8—288 aspreviously disclosed.

Referring to FIG. 9, the primary application consists of one thread T0342 with the interceptor 344. The backup application likewise consistsof one thread B-T0 348 and the resource interceptor 346. The primaryapplication is launched as is the backup application.

The primary thread calls fopen( ) and is intercepted 352. The fopen( )call is processed by the I/O resource (not shown as explained above) andthe return value from (open is packaged into the DATA block and thereplication message METHOD_NONE,S0,P,T0, {DATA,len,data1} is sent 354 tothe backup interceptor 346 via the messaging engine. This is followed byfopen( ) returning 360 to the calling thread 342. On the backup the mainthread B-T0 is processing and reaches fopen( ) 358, which is intercepted356. The interceptor requests the replication message with <P,T0> and isdelivered the matching message S0,P,T0,{DATA,len,data1}. As disclosedpreviously, the backup doesn't open the file, rather it uses the data inthe DATA block to determine the actual return value of fopen( ) and toset the internal state of the FILE object. This is followed by returning362 the return value to the calling thread 348. The backup applicationthus operates under the assumption that it has opened the file, eventhough it has only been presented with the results from the primary.

Later the primary thread 342 calls fwrite( ) 370 which is intercepted372. The write operation is completed using the I/O resource and theresults packed into the DATA block of the replication messageMETHOD_NONE, S1, P,T0,{DATA,len2,data2}. The replication message is sent374 via the messaging engine and eventually retrieved by the interceptoron the backup 376. In the meantime the backup thread is executing andreaches the fwrite( ) 378 call, which is intercepted 376. Theinterceptor requests the replication message corresponding to <P,T0> andis delivered the above mentioned message when available. The data in theDATA block of the replication message is used to set the return value offwrite( ) 380, and to set the internal state of the FILE pointer; noactual write takes place. Upon returning to the main thread in thebackup 348 the program continues under the assumption that a file hasbeen written, even tough no writing took place on the backup.

Finally, the thread T0 342 calls fclose( ) 390, which is intercepted392. The close operation is completed using the I/O resource and theresult packed into the DATA block of the replication messageMETHOD_NONE,S2,P,T0, {DATA,len3,data3}. The replication message is sent394 via the messaging engine and eventually retrieved by the interceptor396 on the backup. This is followed by fclose( ) returning 400 to thecalling thread. In the meantime the backup thread continues executingand calls fclose( ) 398, which is intercepted 396. The interceptorrequest the replication message corresponding to <P,T0> and uses thedata in the data block to set the return value and internal state of theFILE object. Said return value is returned via fclose( )'s return 402.

9.2 I/O Resources—Reading Data

For Read operations the same general technique is used. The primaryapplication is responsible for all reading operations, while the backupreceives a DATA block indicating the read operation results. For readoperations the DATA block additionally contains the actual data read.The data is encoded along with return values and parameters using thepreferred embodiment disclosed above. As with write-operations, andalternate embodiment with custom encoding is also considered.

int main(void)

{

int length=10;

-   -   char pStr[length];

int count=0;

-   -   FILE *fp=fopen(“/home/user/newfile.txt”, “r”)    -   if (fp !=null)        -   count=fread(pStr,1, length,fp);    -   fclose(fp)

}

By way of example, which reads 10(length) characters from a filegenerates the following replication messages

METHOD_NONE, S0, P, T0, {DATA, len1, data1} // fopen( ) METHOD_NONE, S1,P, T0, {DATA, len2, data2} // fread( ) METHOD_NONE, S2, P, T0, {DATA,len3, data3} // fclose( )

The DATA block for fread( ) is the only one which is substantivelydifferent from the previous fwrite( ). For fread( ) the DATA blockencodes the return value (count), the parameter (fp) and the content ofbuffer read (pStr).

Upon retrieving the fread( ) replication message the interceptor forfread( ) on the backup updates the return value (count), updates thestate of the local FILE object and copies the pStr from the DATA blockinto the pStr on the backup. The interceptor then returns the fread( )to the calling thread. On the backup no data is read, rather theoriginal fread( ) is intercepted and suppressed, and the data read bythe primary is supplied to the interceptor which uses it in-lieu ofreading the data.

While in some cases it would be possible to let the backup actually readthe data directly and not pass it via replication messages that is notalways the case. Some storage devices only allow one access at any onetime, some storage device might be mounted for single user access, orthe read operation might actually be from a location in primary localmemory not accessible by the backup.

Similarly, for network read operations using, by way of example, socketsit's only possible to read/receive any particular message once. Thebackup does not have the ability to also read the incoming message.

Thus, in the preferred implementation, data read is passed viareplication messages to the backup. In an alternate implementation, thebackup reads the data wherever possible.

9.3 I/O Resources—Other

For read and write operations that affect system libraries similarteachings apply. By way of example, srand (unsigned int seed)initializes a random number generator with a chosen seed value. This isequivalent to a write operation to “a library memory location” and thecorresponding replication message METHOD_NONE,S0,P0,T0,{DATA,len1,data1}has the seed value encoded within the DATA block. The seed value is thuspassed to the backup.

By way of example, “double rand( )”, which generates a random number issimilar to a read( ) operation in that it produces a number from thesystem library. The corresponding replication message is againMETHOD_NONE,S0,P0,T0,{DATA,len2,data2}. The random number is encoded asthe return value and passed via a replication message to the backup.When the backup program executes the rand( ) method call, it ispresented with the value of rand( ) produced on the primary, and is notgenerating its own.

The general teachings are thus: for write operations the writes areperformed on the primary and the results and parameters are sent to thebackup using replication messages. For read operations the reads areperformed on the primary and the results, parameters and data-read aresent to the backup using replication messages.

10. Deployment Scenarios

FIG. 10 further illustrates by way of example embodiment 420 a varietyof ways the invention can be configured to operate.

In one embodiment, the invention is configured with a central fileserver 422, primary server 424 and backup server 426. The primary server424 runs the primary application and the backup server runs the backupapplication. The primary 424 and backup 426 are connected to each otherand the storage device 422 via a network 428. The network is connectedto the internet 436 for external access. In another embodiment theprimary server 424 is replicated onto two backup servers; backup 426 andbackup-2 425. In yet another embodiment the primary 424 runs in the datacenter, while the backup 427 runs off site, accessed over the internet

In one embodiment a PC client 432 on the local network 428 is connectedto the primary application while the backup application is prepared totake over in the event of a fault. In another embodiment a PC 434 isconfigured to access the primary application server 424 over the publicinternet 436. In a third embodiment a cell phone or PDA 430 is accessingthe primary application 424 over wireless internet 438,436. The presentinvention is configured to server all clients simultaneouslyindependently of how they connect into the application server; and inall cases the backup server is continuously replicating prepared to takeover in the event of a fault

Finally, as the interceptors and messaging engine are componentsimplemented outside the application, the operating system and systemlibraries, the present invention provides replication consistencywithout requiring any modifications to the application, operating systemand system libraries.

The just illustrated example embodiments should not be construed aslimiting the scope of the invention but as merely providingillustrations of some of the exemplary embodiments of this invention

11. Conclusion

In the embodiments described herein, an example programming environmentwas disclosed for which an embodiment of programming according to theinvention was taught. It should be appreciated that the presentinvention can be implemented by one of ordinary skill in the art usingdifferent program organizations and structures, different datastructures, and of course any desired naming conventions withoutdeparting from the teachings herein. In addition, the invention can beported, or otherwise configured for, use across a wide-range ofoperating system environments.

Although the description above contains many details, these should notbe construed as limiting the scope of the invention but as merelyproviding illustrations of some of the exemplary embodiments of thisinvention. Therefore, it will be appreciated that the scope of thepresent invention fully encompasses other embodiments which may becomeobvious to those skilled in the art, and that the scope of the presentinvention is accordingly to be limited by nothing other than theappended claims, in which reference to an element in the singular is notintended to mean “one and only one” unless explicitly so stated, butrather “one or more.” All structural and functional equivalents to theelements of the above-described preferred embodiment that are known tothose of ordinary skill in the art are expressly incorporated herein byreference and are intended to be encompassed by the present claims.Moreover, it is not necessary for a device or method to address each andevery problem sought to be solved by the present invention, for it to beencompassed by the present claims. Furthermore, no element, component,or method step in the present disclosure is intended to be dedicated tothe public regardless of whether the element, component, or method stepis explicitly recited in the claims. No claim element herein is to beconstrued under the provisions of 35 U.S.C. 112, sixth paragraph, unlessthe element is expressly recited using the phrase “means for.”

What is claimed is:
 1. A system for providing replica consistencybetween a primary application and one or more backup applications, thesystem comprising: computer system memory comprising one or more memorylocations configured to store the primary application; one or moreCentral Processing Units (CPUs) operatively connected to said computersystem memory and configured to execute said primary application on aprimary host with a host operating system; an interception layer on theprimary application configured to intercept access to processes andthreads and generate replication messages based on said interception,wherein said intercepted access includes one or more of creatingthreads, destroying threads, creating processes, and destroyingprocesses; a messaging engine on the primary application configured tosend said replication messages to the one or more backup applications;and backup interception layers corresponding to each backup application,said backup interception layers configured to intercept access toprocesses and threads; wherein each instance of creating and destroyingthreads and processes are assigned a unique Method ID, and eachreplication message contains at least the Method ID; whereininterceptors for said each instance of creating and destroying threadsand processes send unique replication messages with said unique MethodIDs to the one or more backups; wherein said unique replication messagesare used on the one or more backup applications to maintain mappingsbetween a primary process ID and thread ID pair and a correspondingprocess ID and thread ID pair on the one or more backup applications;and wherein replica consistency is provided by imposing a same eventordering on each backup application as on the primary application. 2.The system according to claim 1, where said host operating system is oneof Linux, UNIX or Windows.
 3. The system according to claim 1, whereincall ordering is encoded by increasing the sequence number by one onsaid primary application for each new replication message.
 4. The systemaccording to claim 1, wherein the ordering of the primary application'screation and destruction of processes and threads is imposed on the oneor more backup applications by sorting incoming replication messages bysequence number, and only delivering replication messages with matchingMethod ID and with a sequence number exactly one larger than the mostrecently delivered message.
 5. The system according to claim 4, whereinrequests for replication messages without matching Method ID are placedin a pending thread callback queue until an oldest replication messagematches said Method ID.
 6. The system according to claim 1, wherein saidunique replication messages is a replication message with Method ID ofPROCESS_INIT sent prior to the application's main( ) method beingcalled.
 7. The system according to claim 1, wherein said uniquereplication message is a replication message with Method ID ofPROCESS_CREATE sent after creation of a new process, or a replicationmessage with Method ID of THREAD_CREATE sent after creation of a newthread.
 8. The system according to claim 1, wherein said uniquereplication message is a replication message with Method ID ofPROCESS_EXIT sent after termination of a process, or a replicationmessage with Method ID of THREAD_EXIT sent after termination of athread.
 9. The system according to claim 1, wherein a replicationmessage for creating a process further contains a process ID of a parentprocess.
 10. The system according to claim 1, wherein a replicationmessage for creating a thread further contains a process ID of a parentprocess.
 11. The system according to claim 1, wherein sequence numbersare generated as a globally unique identifier for the application. 12.The system according to claim 1, wherein said replication messages aresent from the primary application to the one or more backup applicationover one of TCP/IP or UDP.
 13. The system according to claim 1, whereinsaid interception layer on the primary application first calls a processor thread, then creates and sends a corresponding replication message,and then returns the call to the calling application.
 14. The systemaccording to claim 1, wherein the interceptor on the backup applicationfirst requests a replication message from the messaging engine, thencalls the corresponding process or thread method, and then returns tothe calling application.
 15. The system according to claim 1, whereinthe primary application and the one or more backup applications areconnected over one of a local area network, wide area network, Internetor wireless network.
 16. The system according to claim 1, whereinencoding is performed using one of XML or custom encoding, and whereinthe interception layer and messaging engine are implemented without theneed to modify one of the application, operating system or systemlibraries.
 17. A system for providing replica consistency between aprimary application and one or more backup applications, the systemcomprising: computer system memory comprising one or more memorylocations configured to store the primary application; one or moreCentral Processing Units (CPUs) operatively connected to said computersystem memory and configured to execute said primary application on aprimary host with a host operating system; an interception layer on theprimary application configured to intercept access to processes andthreads and generate replication messages based on said interception,wherein said intercepted access includes one or more of creatingthreads, destroying threads, creating processes, and destroyingprocesses; a messaging engine on the primary application configured tosend said replication messages to the one or more backup applications;and backup interception layers corresponding to each backup application,said backup interception layers configured to intercept access toprocesses and threads; wherein each instance of creating and destroyingthreads and processes are assigned a unique Method ID, and eachreplication message contains at least the Method ID, and replicaconsistency is provided by imposing the same event ordering on eachbackup application as on the primary application; wherein an interceptorof the primary application sends a PROCESS_INIT replication messageprior to the application's main( ) method is executed; and wherein saidPROCESS_INIT replication message is used on the one or more backupapplications to build a mapping between a primary application's initialprocess ID and thread ID pair and the corresponding process ID andthread ID pair on the one or more backup applications.
 18. The systemaccording to claim 17, wherein the initial mapping between the primaryapplication process and thread ID pair and one or more backup processesand thread ID pair is implemented using one of a hash table or adatabase.
 19. A system for providing replica consistency between aprimary application and one or more backup applications, the systemcomprising: computer system memory comprising one or more memorylocations configured to store the primary application; one or moreCentral Processing Units (CPUs) operatively connected to said computersystem memory and configured to execute said primary application on aprimary host with a host operating system; an interception layer on theprimary application configured to intercept access to processes andthreads and generate replication messages based on said interception,wherein said intercepted access includes one or more of creatingthreads, destroying threads, creating processes, and destroyingprocesses; a messaging engine on the primary application configured tosend said replication messages to the one or more backup applications;and backup interception layers corresponding to each backup application,said backup interception layers configured to intercept access toprocesses and threads; wherein each instance of creating and destroyingthreads and processes are assigned a unique Method ID, and eachreplication message contains at least the Method ID, and replicaconsistency is provided by imposing the same event ordering on eachbackup application as on the primary application; wherein the primaryapplication's interceptor sends a PROCESS_CREATE replication messageafter creation of a new process, and a THREAD_CREATE replication messageis sent after creation of a new thread; and wherein said PROCESS_CREATEand THREAD_CREATE replication messages are used on the one or morebackup applications to build a mapping between a primary process ID andthread ID pair and a corresponding process ID and thread ID pair on theone or more backup applications.
 20. A system for providing replicaconsistency between a primary application and one or more backupapplications, the system comprising: computer system memory comprisingone or more memory locations configured to store the primaryapplication; one or more Central Processing Units (CPUs) operativelyconnected to said computer system memory and configured to execute saidprimary application on a primary host with a host operating system; aninterception layer on the primary application configured to interceptaccess to processes and threads and generate replication messages basedon said interception, wherein said intercepted access includes one ormore of creating threads, destroying threads, creating processes, anddestroying processes; a messaging engine on the primary applicationconfigured to send said replication messages to the one or more backupapplications; and backup interception layers corresponding to eachbackup application, said backup interception layers configured tointercept access to processes and threads; wherein each instance ofcreating and destroying threads and processes are assigned a uniqueMethod ID, and each replication message contains at least the Method ID,and replica consistency is provided by imposing the same event orderingon each backup application as on the primary application; wherein theprimary application's interceptor is configured to send a PROCESS_EXITreplication message after termination of a process and a THREAD_EXITreplication message after termination of a thread; and wherein saidPROCESS_(—) EXIT and THREAD_EXIT replication messages are used on theone or more backup applications to remove a mapping between a primaryprocess ID and thread ID pair and a corresponding process ID and threadID pair on the one or more backup applications.